Monitor

ABSTRACT

When a target area is monitored by way of panoramic images of the area that are generated by shooting the target area, sequentially shifting the shooting direction, a unit image of a generated panoramic image and a corresponding unit image of another panoramic image generated before the first panoramic image are compared and the change, if any, in the luminance level is detected. Then, the detected difference of luminance level is compared with a predetermined threshold value and predetermined information is displayed on the display screen along with the generated panoramic image. It is possible to define small regions in the area for which a large threshold is predetermined and/or small regions in the area for which no such comparison is made on a unit image basis.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a monitor for monitoring the situation of abroad area by means of a panoramic image obtained by shooting thesurroundings, sequentially changing the shooting direction and also to amonitoring method, a computer program and a recording medium to be usedwith such a monitor.

This application claims priority of Japanese Patent Application No.2003-415744, filed on Dec. 12, 2003, the entirety of which isincorporated by reference herein.

2. Description of Related Art

Electronic still cameras have been and being popularly used. They areadapted to convert light transmitted through a lens as a result ofshooting a target by means of a solid state image pickup element such asCCD into image signals and record the image signals on a recordingmedium. They can also reproduce the recorded image signals. Manyelectronic still cameras are equipped with a monitor that can display apicked up still image so that a specific one of the recorded stillimages may be displayed there. In such an electronic still camera, theimage signals supplied to the monitor correspond to an image of thetarget of shooting. Thus, it is an image of a very limited area that canbe displayed at a time. In other words, it has not been possible tomonitor the situation of a broad area.

In view of the above identified circumstances, monitoring cameras thatcan continuously shoot a target, sequentially shifting the shootingdirection, to obtain a panoramic view of the target by arranging aplurality of unit images in order to monitor the situation of a broadarea have become popular. Particularly, in recent years, techniques ofreducing the size of each image and synthetically combining a pluralityof images into a single image to produce video signals of a single framehave been proposed (see, inter alia, Patent Document 1: Jpn. Pat. Appln.Laid-Open Publication No. hei 10-108163). Additionally, centralmonitoring/recording systems adapted to monitor a broad area bycollecting images that are obtained by means of a plurality ofmonitoring video cameras arranged at respective positions have also beenproposed (see, inter alia, Patent Document 2: Jpn. Pat. Appln. Laid-OpenPublication No. 2000-243062).

When shooting a target area with a predetermined view angle by means ofa known monitoring camera as shown in FIG. 1 of the accompanyingdrawings, it is necessary to sequentially shift the shooting directionboth horizontally and vertically. When, for example, the size of thetarget area is expressed by s×t times of the frame size of each image(to be referred to as unit image) obtained by shooting a small region ofthe target area from a given view angle, it is necessary to select atleast s×t shooting directions.

To be more specific, the target is shot by aligning the shootingdirection of the monitoring camera to coordinates (1, 1) of the upperleft corner. Then, the shooting direction of the monitoring camera isshifted sequentially and horizontally to coordinates (2, 1), (3, 1), (4,1), . . . , (s, 1) to shoot the target continuously. After completingthe operation of shooting the first row, the target is shot by aligningthe shooting direction of the monitoring camera to coordinates (1, 2) ofthe second row and then shifting the shooting direction sequentially andhorizontally. After repeating the above operation until the shootingdirection is aligned to coordinates (s, t) for shooting the target, thes×t unit images are put together to synthetically produce an image ofthe entire target area.

However, when a watchman tries to detect a change in the target area,using the monitored image recorded on a recording medium such as a videotape in a manner as described above, he has to observe every inch of thetarget area in the monitored image that is updated continuously by eachscanning motion of the camera. Particularly, the watchman has toidentify any minor change in the image and appearance of a small objectat the cost of a large effort and time. Additionally, since such amonitoring system heavily relies on the eyes of the watchman, itinherently involves a risk that changes and appearances of objects canbe overlooked.

Still additionally, when a change is detected in the target area, thewatchman has to analyze the situation and the cause of the change indetail by referring to the image data recorded in the past to furtherraise the load on the part of the watchman.

Furthermore, in a system where any difference of luminance betweenpanoramic images taken with time intervals calls watchman's attentionand the target area is what is shown in FIG. 1, the road leading to theentrance of the church may frequently calls watchman's attention becausemany people may be walking there and hence movements may be detectedalso frequently. If the watchman needs to monitor small regions of thetarget area other than the road where many movements may be detected, hemay be constantly annoyed by the movements on the road that may arise onthe road because they inevitably call his attention. Then, the watchmancan eventually mix up the changes detected on the road and thosedetected in other small regions of the target area so that he may not beable to reliably monitor the target area.

SUMMARY OF THE INVENTION

Therefore, it is an object of the present invention to dissolve theabove identified problems of known monitors of the type underconsideration by providing a monitor that can alleviate the load on thepart of the watchman and detect any change in a broad target areaquickly and accurately, focusing only on a desired small region of thetarget area, along with a monitoring method, a computer program and arecording medium that can be used with such a monitor.

Thus, according to the invention for dissolving the above identifiedproblems, when a target area is monitored by way of panoramic images ofthe area that are generated by shooting the target area, sequentiallyshifting the shooting direction, a unit image of a generated panoramicimage and a corresponding unit image of another panoramic imagegenerated before the first panoramic image are compared and the change,if any, in the luminance level is detected. Then, the detecteddifference of luminance level is compared with a predetermined thresholdvalue and predetermined information is displayed on the display screenalong with the generated panoramic image. It is possible to define oneor more than one small regions in the area for which a large thresholdis predetermined and/or small regions in the area for which no suchcomparison is made on a unit image basis.

More specifically, in an aspect of the present invention, there isprovided a monitor for monitoring a panoramic image generated byshooting a target area, sequentially shifting the shooting direction,the monitor comprising: a comparing means for sequentially detecting thedifference of luminance level between each unit image forming thegenerated panoramic image and the corresponding unit image taken priorto the first unit image along the same shooting direction and comparingthe detected difference of luminance level with a predeterminedthreshold value; a defining means for predefining one or more than oneregions with a threshold value greater than the predetermined thresholdvalue and/or one or more than one regions exempt from the comparison, ona unit image basis; and a display control means for causingpredetermined information to be displayed on a display screen along withthe generated panoramic image according to the result of the comparisonby the comparing means.

In another aspect of the invention, there is provided a monitoringmethod of monitoring a panoramic image generated by shooting a targetarea, sequentially shifting the shooting direction, the methodcomprising: a comparing step of sequentially detecting the difference ofluminance level between each unit image forming the generated panoramicimage and the corresponding unit image taken prior to the first unitimage along the same shooting direction and comparing the detecteddifference of luminance level with a predetermined threshold value; anda display step of causing predetermined information to be displayed on adisplay screen along with the generated panoramic image according to theresult of the comparison by the comparing means; one or more than oneregions with a threshold value greater than the predetermined thresholdvalue and/or one or more than one regions exempt from the comparisonbeing predefined on a unit image basis in the comparing step.

In still another aspect of the invention, there is provided a computerprogram for causing a computer to monitor a panoramic image generated byshooting a target area, sequentially shifting the shooting direction,the program comprising: a comparing step of sequentially detecting thedifference of luminance level between each unit image forming thegenerated panoramic image and the corresponding unit image taken priorto the first unit image along the same shooting direction and comparingthe detected difference of luminance level with a predeterminedthreshold value; and a display step of causing predetermined informationto be displayed on a display screen along with the generated panoramicimage according to the result of the comparison in the comparing step;one or more than one regions with a threshold value greater than thepredetermined threshold value and/or one or more than one regions exemptfrom the comparison being predefined on a unit image basis in thecomparing step.

In a further aspect of the present invention, there is provided arecording medium storing a computer program for causing a computer tomonitor a panoramic image generated by shooting a target area,sequentially shifting the shooting direction, the program comprising: acomparing step of sequentially detecting the difference of luminancelevel between each unit image forming the generated panoramic image andthe corresponding unit image taken prior to the first unit image alongthe same shooting direction and comparing the detected difference ofluminance level with a predetermined threshold value; and a display stepof causing predetermined information to be displayed on a display screenalong with the generated panoramic image according to the result of thecomparison in the comparing step; one or more than one regions with athreshold value greater than the predetermined threshold value and/orone or more than one regions exempt from the comparison being predefinedon a unit image basis in the comparing step.

Thus, according to the invention, when a target area is monitored by wayof panoramic images of the area that are generated by shooting thetarget area, sequentially shifting the shooting direction, a unit imageof a generated panoramic image and a corresponding unit image of anotherpanoramic image generated before the first panoramic image are comparedand the change, if any, in the luminance level is detected. Then, thedetected difference of luminance level is compared with a predeterminedthreshold value and predetermined information is displayed on thedisplay screen along with the generated panoramic image. It is possibleto define one or more than one small regions in the area for which alarge threshold is predetermined and/or small regions in the area forwhich no such comparison is made on a unit image basis.

With the above-described arrangement, watchman's attention is no longercalled unnecessarily and frequently and the watchman can focus on thesmall regions of the area where any movement has to be detected to raisethe efficiency and the accuracy of the monitoring operation.

BIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a picked up panoramic whole image;

FIG. 2 is a schematic illustration of a monitoring system realized byapplying the present invention;

FIG. 3 is a schematic block diagram of a monitoring system realized byapplying the present invention, showing the configuration thereof;

FIG. 4 is a schematic block diagram of a camera unit and a monitoraccording to the invention;

FIG. 5 is a schematic illustration of an operation of a camera unit ofshooting a target area defined by a black frame with a view angle u;

FIG. 6 is a schematic illustration of an image on a display, showing theelements of the image;

FIG. 7 is a schematic illustration of the operation of a monitoringsystem realized by applying the present invention;

FIG. 8 is a schematic illustration of an operation of comparing unitimages for luminance level, using an image compression technique;

FIG. 9 is a schematic illustration of a panoramic image of a targetarea, showing unit images where a difference is detected in shade as aresult of comparing the luminance levels on a unit image basis;

FIG. 10 is a schematic illustration of a panoramic image of a targetarea, showing the small image regions calling attention where adifference is detected as a result of comparing the luminance levels ona unit image basis;

FIG. 11 is a schematic illustration of unit images for which a thresholdvalue has already been defined and in which different threshold valuesare defined respectively for desired small image regions;

FIG. 12 is a schematic illustration of a panoramic image for whichthreshold values are defined and stored for specified desired smallimage regions;

FIG. 13 is a schematic illustration of a technique of defining a colorlevel and a difference detection level for each primary color;

FIG. 14 is a schematic illustration of a technique of selecting the sizeof the object for which difference is detected;

FIG. 15 is a schematic illustration of a technique of comparing the sizeof a detected small region and that of a reference image region; and

FIG. 16 is a flow chart illustrating the sequence of operation ofproviding information on the detected difference to a user operating aterminal unit.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Now, the present invention will be described in greater detail byreferring to the accompanying drawings that illustrate preferredembodiments of the invention. Firstly, referring to FIG. 2 thatschematically illustrates a monitoring system realized by applying thepresent invention, the monitoring system 1 comprises a camera unit 2 forshooting a target and generating image signals, a monitor 5 to which theimage signals are transmitted, a display 6 connected to the monitor 5, aterminal unit 9 where a plurality of users execute application programs,a terminal display 10 connected to the terminal unit 9 and a network 8for realizing bidirectional communications between the camera unit 2 andthe monitor 5 and between the camera unit 2 and the terminal unit 9.

The camera unit 2 of this monitoring system 1 by turn comprises apan-tilter section 3 and a camera section 4 as integral componentsthereof as shown in FIG. 3. The pan-tilter section 3 is a rotary tablefor freely changing the shooting direction along, for example, two axesof a panning axis and a tilting axis.

The camera section 4 is mounted on the rotary table of the pan-tiltersection 3 and adapted to shoot a target of imaging under the control themonitor 5, adjusting its shooting direction horizontally and/orvertically. Additionally, the camera section 4 increases or decreasesthe shooting magnification when shooting the target under the control ofthe monitor 5, while sequentially shifting the shooting direction. It ispossible to acquire multilateral image information by arranging aplurality of such camera sections 4 for a single monitor 5 and shootingthe same target from different shooting angles.

Referring to FIG. 3, the monitor 5 comprises an image input/outputsection 13 for conducting a predetermined processing operation on theimage signals transmitted from the camera unit 2, a processing section15 connected to the image input/output section 13 to generate a movingimage according to the image signals, a server 53 connected to theprocessing section 15 to record the image signals and an operationsection 16 to be used by the user to control the monitor 5.

The monitor 5 is typically constituted by an electronic device such aspersonal computer (PC) and adapted to record the image signalstransmitted from the camera unit 2 and display images on the display 6for the user according to the recorded image signals. Additionally, whenthe user specifies a desired image region or a desired image position,the monitor 5 controls the operation of selecting an image thatoptimally meet the specification from the recorded image signals anddisplays it. Thus, the monitor 5 takes the role of central processingunit that controls the entire network 8 and transmits images in responseto a request from some other terminal unit 9. The monitor 5 will bedescribed in greater detail hereinafter.

The network 8 is a public communication network to which the monitor 5is connected by way of a telephone line and which allows bidirectionaltransmission/reception of information such as the Internet or an ISDN(integrated services digital network)/B (broadband)−ISDN. If themonitoring system 1 is operated in a small area, the network 8 may be aLAN (local area network). Additionally, the network 8 may be adapted totransmit MPEG images in addition to still images. If such is the case,based on the Internet Protocol (IP) MPEG data will be transmittedcontinuously by way of a channel, while still image data will betransmitted periodically by way of another channel.

The terminal unit 9 is a PC by means of which the user who may be athome or in the office acquires images from the monitor 5 by way of thenetwork 8 and execute desired processing operations. When a plurality ofterminal units 9 are connected to the network 8, the application programof the monitoring system 1 can be provided to the plurality of userssimultaneously. The terminal unit 9 displays the images acquired formthe monitor 5 on the terminal display 10. Additionally, the terminalunit 9 generates a request signal according to an instruction-givingoperation of the user and transmits it to the monitor 5. Since theconfiguration of the terminal unit 9 is similar to that of the monitor5, which will be described hereinafter, and hence will not be describedhere any further.

Now, the camera unit 2 and the monitor 5 of the monitoring system 1realized by applying the present invention will be described below indetail.

FIG. 4 illustrates the configuration of the camera unit 2 and that ofthe monitor 5 in detail. In FIG. 3, the components of the camera unit 4and those of the monitor 5 are connected to a common controller bus 21.

The pan-tilter section 3 of the camera unit 2 includes a tilt section 3a and a pan section 3 b for controlling the rotary table in order tochange the shooting direction. The camera section 4 of the camera unit 2includes a lens control section 23 for mainly changing the view angle ofthe lens section 22, an image pickup section 24 arranged at a positionperpendicularly intersecting the optical axis of the lens section 22, anIEEE1394 (Institute of Electrical and Electronics Engineers) 1394interface 25 for transmitting the image signals generated by the imagepickup section 24 to the image input/output section 13, a GPS (globalpositioning system) receiving section 28 for detecting the currentposition of the camera unit 2 and a meta-data generating section 29fitted to the GPS receiving section 28. Note that the IEEE1394 interface25 may be replaced by Ethernet®.

The image input/output section 13 has a buffer memory 51 that isconnected to the IEEE1394 interface 25 and an encoder 52 that isconnected to the buffer memory 51.

The processing section 15 has to an image compressing section 54 forcompressing the images read out from the server 53, a graphic controller55 connected to the server 53 and the image compressing section 54 togenerate images to be displayed on the display 6, a CPU 56 forcontrolling the other sections of the camera unit 2 by way of thecontroller bus 21 and a memory card 61 and a clock 62 that are connectedto I/O port 58.

The operation section 16 has a keyboard 59 and a mouse 60 to be used bythe user to specify a desired image region and a desired image positionin the image being display on the display 6.

The tilt section 3 a and the pan section 3 b drive the stepping motorthat operates as the drive source of the rotary table according to thedrive signal from the CPU 56. As a result, the shooting direction of thecamera section 4 mounted on the rotary table can be changed horizontallyand/or vertically.

The lens control section 23 performs an automatic aperture controloperation and an automatic focus control operation on the lens section22 according to the drive signal from the CPU 56. Additionally, the lenscontrol section 23 changes the shooting direction relative to the targetaccording to the drive signal. As a result, the camera section 4 cansequentially regulate the magnification when shooting the target.

The image pickup section 24 comprises a solid state image pickupelement, which may typically be a CCD (charge coupled device), and isadapted to form an image of the target on the imaging plane of the imagepickup element by focusing rays of light coming in through the lenssection 22, generate image signals by photoelectric conversion andtransmit the generated video signals to the IEEE1394 interface 25.

The GPS receiving section 28 detects the position and the shootingdirection of the camera unit 2 according to the signal transmitted fromthe GPS system. Due to the provision of the GPS receiving section 28,particularly when a plurality of camera units 2 are arranged, it ispossible to interlock the shooting directions of the camera units 2 in acontrolled manner. The output signal of the GPS receiving section 28 issupplied to the meta-data generating section 29, which generatespositional information including the latitude, the longitude, theazimuth and the altitude of the camera unit 2 and meta-data includingthe current clock time and various parameters. The meta-data generatingsection 29 supplies the positional information and the meta-data itgenerates to the encoder 52. For the purpose of the present invention,the GPS receiving section 28 and the meta-data generating section 29 maybe omitted.

The buffer memory 51 temporarily stores the image signals supplied fromthe IEEE1394 interface 25 according to the control signal from the CPU56. The video signals that are temporarily stored in the buffer memory51 are then supplied to the encoder 52 and subjected tocompression/coding according to the applicable standard such as the JPEG(Joint Photographic Experts Group) Standard. Note that the encoder 52may add the positional information and the meta-data to the imagesignals to be compressed/coded that are supplied from the meta-datagenerating section 29. The encoder 52 outputs the compressed/coded imagesignals to the server 53 or the image compressing section 54. When thesupplied image signals are not subjected compression/coding, theoperation of the encoder 52 will be omitted.

The server 53 sequentially records the image signals output from theencoder 52 after correlating them with the positional information andthe meta-data. Note that the server 53 may be replaced by a hard disc ora disc-shaped recording medium that can removably fitted to the cameraunit 2. The image signals recorded in the server 53 are read to theimage compressing section 54 and the graphic controller 55 under thecontrol of the CPU 56. If the image signals recorded in the server 53are also recorded on the memory card 61 under control, the user cantransfer the images he or she has picked up to some other PC. If theimage signals recorded in the server 53 are recorded in a network server(not shown) under control, the server 53 may be replaced by the networkserver (not shown).

The image compressing section 54 generates compressed images orthumbnail images for the video signals in the JPEG format that are readout from the server 53. The image compressing section 54 also reads outthe images recorded in the server 53 and generates a moving image underthe control of the CPU 56. The technique according to the MPEG,Motion-JPEG or Motion-JPEG2000 may be used as compression technique whengenerating such a moving image.

The graphic controller 55 performs an image processing operation forgenerating the image to be displayed on the display 6 according to theimage signals read out from the server 53 or the image signals outputfrom the image compressing section 54. The graphic controller 55 alsocontrols the contrast and the luminance of the image being displayed onthe display 6 under the control of the CPU 56.

When the user specifies an image region and an image position by way ofthe keyboard 59 and/or the mouse 60, the CPU 56 transmits a drive signalfor driving the pan-tilter section 3 and the lens control section 23 anda control signal for controlling the component sections of the monitor 5by way of the controller bus 21. Additionally, when the CPU 56 receivesa predetermined request signal from the terminal unit 9, it controls theoperation of the camera unit 2 so as to select an optimal still image,an optimal moving image or an optimal piece of information recorded inthe server 53 and transmit it to the terminal unit 9.

Now, the shooting operation of the monitoring system 1 realized byapplying the present invention will be described below.

FIG. 5 is a schematic illustration of an operation of a camera unit 2 ofshooting a target area defined by a black frame with a view angle u. Theshooting direction needs to be sequentially shifted both horizontallyand vertically in order to shoot the entire target area with a viewangle u. If the size of the entire target area is i×j times of the framesize of the image (to be referred to as unit image hereinafter) obtainedby shooting the target with a view angle u, it is necessary to select atleast i×j shooting directions. An image of the entire target area can besynthetically obtained by putting the i×j unit images obtained byshooting the target with the view angle u together.

If the coordinates (M, N) of the unit images of the target area takevalues of 1, 2, . . . , M, . . . , i from the left to right in thehorizontal direction and 1, 2, . . . , N, . . . , j from top to down inthe vertical direction, the CPU 56 firstly transmits a drive signal tothe tilt section 3 a and the pan section 3 b to drive the camera section4 so as to align the shooting direction to coordinates (1, 1) and shootthe target. The image signal of the unit image obtained by shooting thetarget with a shooting direction aligned to coordinates (1, 1) istemporarily stored in the buffer memory 51 and compressed and coded bythe encoder 52 according to the JPEG Standard. Then, the positionalinformation showing the shooting direction and the meta-data transmittedfrom the GPS 28 are added to the image signal and recorded in the server53.

Similarly, the CPU 56 transmits a drive signal to the tilt section 3 aand the pan section 3 b to drive the camera section 4 so as to shift theshooting direction by an image frame, align it to coordinates (2, 1) andshoot the target. The image signal of the unit image obtained byshooting the target with a shooting direction aligned to coordinates(2, 1) is also recorded in the server 53. In this way, the camerasection 4 sequentially shifts the shooting direction horizontally tocoordinates (3, 1), (4, 1), . . . , (i, 1) and shoots the target underthe control of the CPU 56.

After completing the operation of shooting the first row, the camerasection 4 aligns the shooting direction to coordinates (1, 2) of thesecond row and shoots the target under the control of the CPU 56.Subsequently, it sequentially shifts the shooting directionhorizontally. Then, it repeats the above operation in a similar manneruntil it gets to coordinates (i, j). When the camera section 4 completesthe shooting operation, the server 53 stores image signals of the i×junit images.

Note that the image signals of the unit images recorded in the server 53are sequentially read out by the image compressing section 54 and theunit images are dimensionally reduced so as to match the size of thedisplay screen of the display 6 as a whole. The dimensionally reducedunit images are displayed on the display 6 by way of the graphiccontroller 15. As all the i×j unit images recorded in the server 53 aredisplayed on the display 6, a single panoramic image of the entiretarget area is synthetically produced. The above described targetshooting operation is repeated at regular intervals to make it possibleto acquire the most updated image of the entire target area.

FIG. 6 is a schematic illustration of an image on the display 6 thatincludes an image of the entire target area obtained by syntheticallycombining a total of i×j unit images and displayed in a whole imagedisplay section 70 of the display 6. Note that the monitor 5 may havethe whole image display section 70 display the boundaries of the unitimages in the whole image or only the whole image seamlessly.Additionally, the monitor 5 may have the whole image display section 70display a single whole image of the target area that is obtained byshooting the target area with a view angle good for picking up an imageof the entire target area, as substituted for the single panoramic imageof the entire target area.

The display screen 45 also has an enlarged image display section 71 fordisplaying an enlarged unit image. The enlarged image display section 71may display the single unit image specified by the user out of the unitimages of the whole image being displayed in the whole image displaysection 70 or a moving image obtained by shooting the target in theshooting direction of the specified single unit image. With thisarrangement, the user can see the target as viewed in the shootingdirection of the specified single unit image on a real time basis.

The user can specify a desired image region and a desired image positionby way of the keyboard 59 and/or the mouse 60 to the whole image displaysection 70 and the enlarged image display section 71. The sections 70and 71 may display a line of sight and/or a pointer for performing theabove specifying operation in conjunction with the movement of the mouse60 and the like.

The user can select any of various modes of operation of the camera unit2 on the display screen 45 by inputting an instruction by means of themouse 60. For example, the user can increase or decrease themagnification for the unit image being displayed on the enlarged imagedisplay section 71 and control and adjust the shooting direction of thecamera section 4 horizontally and/or vertically.

As the difference detection button 84 being displayed on the displayscreen 45 is clicked by way of the mouse 60, the difference of luminancelevel between a unit image of the generated whole image and thecorresponding unit image obtained before the specified unit image byshooting the target in the same shooting direction is detected at thedifference detection time, which will be described in greater detailhereinafter.

The operation of detecting the difference is conducted by comparing theluminance level of a whole image formed by unit images recorded in theserver 53 or the CPU 56 and that of a reference whole image. The wholeimage for which the difference is detected is referred to as comparativewhole image and the whole image to be used as reference is referred toas reference whole image hereinafter.

Referring now to FIG. 7A, if the reference whole image is whole image a1and the comparative whole image is whole image a2, the difference isdetected for the whole image a2 by comparing the luminance level of eachof the unit images of the whole image a2 with that of the correspondingunit image of the whole image a1. More specifically, the luminance levelof each of the unit images located at respective coordinates (1, 1), (2,1), (3, 1), . . . , (M, N) of the whole image a2 is compared with thatof the corresponding one of the whole image a2 that is obtained byshooting the target in the same shooting direction as shown in FIG. 7A.By doing so, the change in the luminance level of the comparative wholeimage relative to the reference whole image can be detected for eachshooting direction.

When the CPU 56 compares, for example, the luminance level of the unitimage located at coordinates (M, N) of the comparative whole image(whole image a2) with that of the corresponding unit image of thereference whole image (whole image a1), it firstly reads out the twounit images from the server 53. Then, the CPU 56 compares the luminancelevel of each pixel (m, n) located at the same position of one of theunit images and that of the corresponding pixel of the other unit imageas shown in FIG. 7B. The difference of between the two luminance levelsof the two pixels (m, n) located at the same position may be determinedwhen comparing the luminance levels. The CPU 56 can detect thedifference of the luminance levels of the read out unit images by way ofthe differences of the luminance levels of the pixels of the unitimages. Note that the comparison of luminance level between each pair ofpixels may be conducted for all the pixels of unit image or for part ofthe pixels of unit image.

When another whole image a3 is supplied, the whole image a3 takes therole of comparative whole image and the whole image a2 takes the role ofreference whole image and the two whole images are compared for thedifference of luminance level in the same way. Similarly, when stillanother whole image a4 is supplied, the whole image a4 takes the role ofcomparative whole image and the whole image a3 takes the role ofreference whole image and the two whole images are compared for thedifference of luminance level in the same way. In this way, the wholeimages generated by way of the camera section 2 are sequentiallyspecified as comparative whole images and the whole images generatedimmediately before are specified as reference whole images and each pairof whole images are compared for the difference of luminance level tomake it possible to detect any movement in the most updated whole imageinstantaneously.

When comparing the luminance levels of each pair of unit images, the CPU56 may compare them not on a pixel by pixel basis but on an imagecompression unit by image compression unit basis as shown in FIG. 8.

Then, the CPU 56 operates to cut out macro-blocks of 16×16 pixelslocated respectively at the same positions from the unit images withcoordinates (M, N) and compare the luminance levels of the cut outmacro-blocks. Note that, the luminance level of each macro-block isexpressed by the average of the luminance values of 16×16 pixels of themacro-block.

If the luminance levels are compared by using blocks having a sizesmaller than 8×8 pixels, noises are inevitably involved when the imagesignals that have been subjected to compression/coding according to theJPEG Standard are processed for elongation. As a result, it will not bepossible to detect any difference of luminance accurately. In otherwords, the influence of noises can be reduced by comparing the luminancelevels of blocks having a size not smaller than 8×8. Thus, in a monitor5 according to the invention, it is possible to highly accurately detectthe difference of luminance by comparing the luminance levels of sets ofpixels that correspond to the image compression unit of DCT blocks ormacro-blocks.

If, for instance, a difference of luminance level is detected atcoordinates (1, 2), (3, 2) and (4, 3) as a result of comparing theluminance levels on a unit image basis, only the unit images showing adifference of luminance level are displayed with a colored frame to drawattention on the part of the user as shown in FIG. 9. In the displayscreen 45, in addition to displaying colored frames, it may be soarranged as to display each pair of unit images that shows a differenceof luminance level in order to directly notify the user of occurrence ofsuch a difference.

The arrangement for displaying differences of luminance level in themonitor 5 is not limited to the instance of FIG. 9. It may alternativelybe so arranged that, when differences are detected in the luminancelevel as a result of comparing on a pixel basis or on the basis of imagecompression unit of unit image, only the pixels or blocks that showdifferences are displayed with a color to draw attention on the part ofthe user as shown in FIG. 10. With this arrangement, it is possible tonotify the user of any movement detected at a fine pitch.

A predefined threshold value may be used as criterion for determining ifa difference is detected or not. While the CPU 56 detects thedifference, if any, between the luminance levels of the unit images itreads out by means of the difference of luminance level of pixels orblocks, it may be so arranged as to say that a difference is detectedwhen the detected difference exceeds a predefined threshold value. It ispossible to detect movements in a target scene, focusing only on themovements of a desired object in the scene, by predefining an optimumthreshold value depending on the target of shooting and the shootingenvironment.

The same threshold value may be predefined to all the unit images ordifferent threshold values may be predefined respectively to all theunit images. Furthermore, if a threshold value is predefined for a unitimage, different threshold values may be defined for the respectiveimage regions that constitute the image as shown in FIG. 11.

FIG. 11A illustrates an example where threshold value a1 is defined fora whole image and additionally another threshold value a2 is defined fordesired shaded regions β1 in the unit image. Note that the thresholdvalue α2 is larger than the threshold value a 1 in this instance. Thismeans that, if a difference of luminance level that exceeds thethreshold value α1 is detected in the regions β1 for which the thresholdvalue α2 is defined, it is determined that no difference is detected ifthe difference of luminance level does not exceeds the threshold valueα2. In other words, no difference of luminance level is detected if thedetermined difference of luminance level does not exceed the thresholdvalue α2 that is larger than the threshold value α1.

Thus, the regions β1 for which the threshold value α2 is definedscarcely attract attention on the part of the user if a large value isselected for the threshold value α2. An image region in which movementsare relatively frequently observed may draw user's attentionexcessively. However, such an image region can be made to draw user'sattention less by defining the image region as region β1. If the userwants to detect even a slight movement on the church building shown inthe panoramic image of FIG. 10, the surroundings of the road leading tothe entrance of the church that may frequently be crowded by people mayby turn frequently draw attention on the part of the user becausemovements will be detected there frequently. However, if such an imageregion is defined as region β1 and the threshold value α2 is assigned tothe image region, the user may be relieved of a situation where theimage region draws his or her attention frequently and excessively.Then, the user can monitor the image regions that are more important tohim or her for detecting movements accurately and efficiently.

The threshold value α2 may be empirically determined depending on theluminance level of the one ore more than one image regions where anumber of movements are detected per unit time. As for a place that iscrowded by many walking people, the luminance level of a person may beidentified in advance and a threshold value that does not allow personsto be detected may be selected for the threshold value α2 in order toavoid the place from unnecessarily attracting user's attention.Alternatively, the threshold value α2 may be replaced by a number ofdifferent threshold values. Then, the threshold values may be graduallydifferentiated according to the frequency of movement and the luminancelevels of different objects in the scene.

Instead of selecting a large value for the threshold value α2 relativeto the threshold value α1 as described above, the sensitivity ofdetecting any difference of luminance level in the regions β1 may bereduced or the luminance level of the regions β1 may be reduced.

For the purpose of the present invention, regions where the operation ofdetecting any difference of luminance level is not conducted may bedefined in a unit image for which the threshold value α1 is defined.

FIG. 11B shows a unit image for which the threshold value α1 is definedand in which shaded image regions β1 and shaded image regions β2 aredefined. The shaded regions β2 are image regions where the operation ofdetecting any difference of luminance level is not conducted.

Thus, any difference of luminance level is detected for each pixel oreach block in the regions β2 where the operation of detecting anydifference of luminance level is not conducted. In other words, if somemovement takes place and there arises a change in the luminance level ofone or more than one pixel in any of the regions β2, they do not drawany user's attention. If the threshold value α1 is defined for an entireunit image, only one or more than one regions β2 may be defined in theunit image without defining any region β1.

Thus, the monitor 5 of this embodiment can be so arranged that it doesnot detect any difference of luminance level in one or more than oneregions of a unit image where movements take place actively. Therefore,it neither unnecessarily nor frequently draws user's attention in suchimage regions so that the user can monitor the scene highly efficientlyand accurately.

The user defines the threshold value(s) for detecting the difference ofluminance level in the above described monitoring system 1. However, thepresent invention is by no means limited thereto. It may alternativelybe so arranged that a monitoring system realized by applying the presentinvention automatically discriminates one or more than one image regionswhere movements take place frequently and specifies those discriminatedimage regions as regions βor regions β2.

In such a case, the monitor 5 preliminarily shoots the same target areafor a plurality of times before detecting any difference of luminancelevel. The view angle u and the shooting directions of the camera unit 2may be defined in the above described manner. Thereafter, it detects anydifference of luminance level between two unit images picked up from thesame shooting direction.

Then, the monitor 5 specifies each of the image regions where adifference of luminance level is detected either as region β1 or regionβ2. If a difference of luminance level is detected or not is judged bycomparing the difference of luminance level with a threshold value andthe user can define an appropriate threshold value. When the monitor 5specifies one or more than one image regions where a difference ofluminance level is detected as so many regions β1, it is possible todefine an appropriate threshold value α2 on the basis of the magnitudeof the detected difference of luminance level.

In other words, a monitoring system 1 realized by applying the presentinvention automatically defines a threshold value α2 for image regionscontaining one or more than one places where movements take placerelatively frequently or does not conduct any operation of detecting adifference of luminance level when monitoring a desired target area.Then, as a result, it is possible to reduce the load applied to the userof discriminating image regions containing one or more than one placeswhere movements take place relatively frequently by him- or herself byway of a displayed image and specifying them as regions β1 or regionsβ2. Additionally, the monitoring system 1 can automatically andaccurately detect a slight difference of luminance level that the usercan hardly visually discriminate by way of a displayed image and definean optimum threshold value α2.

Note that the number of times for which the monitor 5 preliminarilyshoots the same target area and time intervals of shooting the targetarea may be appropriately selected depending on the shootingenvironment. For example, the monitor 5 may preliminarily shoot the sametarget area three times and determine the logical sum of the detecteddifferences of luminance level. Then, the monitor automaticallyspecifies the image regions expressed by the logical sum of the detecteddifferences of luminance level as regions β1 or regions β2.

Additionally, a monitoring system 1 realized by applying the presentinvention may be so arranged as to store the regions β1 and the regionsβ2 specified in the above described manner by means of the followingtechnique.

Assume that an image position is specified by means of the mouse 60 inthe whole image being displayed on the display screen 45 as illustratedin FIG. 12. Then, as the user points the image position by means of themouse 60, there is displayed a mask defining window 80 including a setof unit images surrounding the image position (to be referred to asgroup of unit images hereinafter). Then, the user can specify desiredimage regions in the displayed group of unit images as regions β1 orregions β2. Or, the user can automatically specify desired image regionsin the displayed group of unit images as regions β1 or regions β2automatically for a plurality of preliminarily picked up images. Then,the user can visually recognize the detected differences of luminancelevel in the group of unit images by means of the mask defining window80. Note that the specified regions β1, β2 and the specified thresholdvalues α1, α2 can be correlated with the group of unit images and storedin the memory (not shown) arranged in the CPU56.

The regions β1, β2 and the threshold values α1, α2 that are stored inthe memory (not shown) will be read out according to the order of theuser who may want to analyze them for the same group of unit images as apost-processing operation. More specifically, if the user wants todetect differences of luminance only in some of the groups of unitimages that constitute the whole image, it is only necessary to read outthe regions β1, β2 and the threshold values α1, α2 of the relatedregions stored in the memory (not shown). Therefore, the work load ofdefining regions and threshold values for such an analysis can bereduced.

A plurality of image positions may be specified in the whole image.Then, it is possible to define the regions β1, β2 and the thresholdvalues α1, α2 for each group of unit images surrounding each of thespecified image position and store them in the memory, correlating themwith the group of unit images. With this arrangement, it is possible toadvantageously detect differences of luminance level for each group ofunit images of the whole image, while reducing the workload on the partof the user.

It may be so arranged that the user can select an appropriate number ofunit images for each group of unit images surrounding an image position.

The monitoring system 1 may be so arranged that the unit images of eachselected group of unit images are newly picked up with the same viewangle u. With such an arrangement, the CPU 56 transmits a drive signalto the tilt section 3 a and the pan section 3 b so as to align theshooting angle of the camera section 4 with the selected group of unitimages for the image pickup operation.

Then, if the user wants to detect differences of luminance only in someof the groups of unit images that constitute the whole image, theworkload of the image pickup operation is minimized to remarkablyimprove the processing speed. Then, it is possible to accurately detectany movement at a fine pitch in the minimally necessary unit images.

The monitor 5 may be so arranged that it compares the luminance levelsof unit images according to the R, G, B primary color signals of theunit images.

FIG. 13A is a schematic illustration of a technique of defining a colorlevel and a difference detection level for the primary color of R (red).In FIG. 13A, the vertical axis represents the difference of luminancelevel (brightness) as computed for each pixel of the unit images to becompared. FIG. 13B shows the maximum value of the differences shown inFIG. 13A in terms of the horizontal direction, while FIG. 13C shows themaximum value of the differences shown in FIG. 13A in terms of thevertical direction.

Referring to FIG. 13, the threshold value L1 and the color level L2 canbe freely defined by the user by means of the keyboard 59 and the mouse60. A difference is detected when the computed difference of luminancelevel of the primary color(R) exceeds the threshold value L1. The colorlevel L2 is used to define the gain of a computed difference in order toimprove the S/N ratio. For example, as a difference value is amplifiedto the color level L2, the remaining differences are also amplified. Itis possible to highly accurately detect differences by defining adesired threshold value L1 for each amplified difference value. Morespecifically, the user can freely define the color level L2corresponding to the level of difference value that the user wants touse for detecting differences and also the extent of change in the levelrelative to the color level L2 to be used as criterion for detectingdifferences by defining the threshold value L1 relative to the colorlevel L2. Note that L1 and L2 can be defined for the remaining primarycolors (G and B).

Note that the threshold value L1 and the color level L2 may bedifferentiated among the primary colors of R, G and B and/or among theunit images of the whole image.

Thus, when the difference of luminance level computed for each primarycolor for the unit images to be compared exceeds the threshold value L1of at least the related one of the three primary colors, the monitoringsystem 1 can determine that a difference is detected for the unitimages. In other words, the monitoring system 1 can detect any suchdifference for each of the primary colors of R, G and B. Then, it ispossible to highly accurately detect any fine change that cannot bevisually detected in the whole image actually displayed on the displayscreen 45.

It is possible to effectively prevent excessively drawing user'sattention by optimally defining the threshold values α2 respective forthe defined threshold values L1.

Note that the threshold value α2 may be differentiated among the primarycolors of R, G and B. Then, it is possible to reduce the frequency ofdrawing user's attention to apparent differences generated by R, G, Bsignals that cannot be visually detected.

Similarly, the monitoring system 1 may be so arranged that the luminancelevels of the unit images to be compared are actually compared accordingto the luminance signal (Y signal) and the color difference signals (U,V signals). With such an arrangement, it is possible to extractinformation on the difference that can be detected on the basis theluminance signal or the color difference signals, although it cannot bedetected on the basis of the primary color signals in a manner asdescribed above, to attract user's attention.

In the above-described monitoring system 1 realized by applying thepresent invention, it is also possible to select the size of the objectfor which any difference of luminance level can be detected.

Referring to FIG. 14A, the user can define a desired reference imageregion in each unit image by means of the keyboard 59 and the mouse 60.Either a single type of reference image region or a plurality ofdifferent type of reference image regions may be defined in each unitimage. Reference image regions may be defined with different sizes forthe primary colors.

Assume that the user defines a reference image region having ahorizontal length (width) of w and a vertical length (height) of t asshown in FIG. 14B. The CPU 56 detects a region showing a differenceexceeding the threshold value L1 in the unit image where the referenceimage region is defined. Then, the CPU 56 compares the size of thedetected region and that of the reference image region.

If the detected region showing a difference exceeding the thresholdvalue L1 is region R1 having a horizontal length (width) of w1 and avertical length (height) of t1 as shown in FIG. 15, the CPU 56 comparesthe horizontal length (width) w1 and the vertical length (height) t1respectively with the horizontal length (width) w and the verticallength (height) t of the reference image region. If the size of thedetected region R1 exceeds that of the reference image region bothhorizontally and vertically, the CPU 56 determines that a difference isdetected. If the size of the detected region R1 falls under that of thereference image region either horizontally or vertically, the CPU 56determines that no difference is detected.

Similarly, if the CPU 56 detects a region showing a difference exceedingthe threshold value L1, which is region R2 having a horizontal length(width) of w2 and a vertical length (height) of t2, the CPU 56 comparesthe horizontal length (width) w2 and the vertical length (height) t2respectively with the horizontal length (width) w and the verticallength (height) t of the reference image region to determine if adifference is detected or not.

Thus, the monitoring system 1 realized by applying the present inventioncan select a region to be used for detecting a difference according tothe reference image region defined by the user. Particularly, nodifference will be detected from a very small region detected by the CPU56 if the size of the reference image region is finely regulated. Then,as a result, the unit image containing such a region is no longerdisplayed with a colored frame to prevent excessively drawing user'sattention.

An operation of selecting a particular object is selected out of thetarget area and detecting any difference of luminance level in it can berealized by defining an appropriate size for a reference image region asa function of the size of the region of the selected object. A wholeimage obtained by shooting the target area, sequentially shifting theshooting direction, normally contains various objects. However, anydifference of luminance level can be detected from each of such objectsby means of a single monitoring system by defining reference imageregions whose sizes may differ from unit image to unit image.

A pair of identical objects may be displayed with different sizes in aunit image depending on the distance from the camera section 4. Forexample, if there are two automobiles of the same type parking ondifferent roads, one parking on a road located remote from the camerasection 4 and the other parking on another road located close to thecamera section 4, the latter is displayed larger than the former in theunit images. However, any difference of luminance level can be detectedaccurately from each of the automobiles of the same type regardless ofthe distance from the camera section 4 when the sizes of the definedreference image regions are altered between the unit image containingthe road located remote from the camera section 4 and the unit imagecontaining the road located close to the camera section 4.

The monitoring system 1 realized by applying the present invention canprovide information on the detected differences to the user who isoperating at the terminal unit 9 by following the flow chart of thesequence of operation illustrated in FIG. 16.

Firstly, in Step S61, the user operating at the terminal unit 9 accessesthe monitor 5 that is connected to the network 8.

Then, in Step S62, the monitor 5 prepares a publishable image list thatcan be disclosed to the user out of the whole images recorded in its ownserver 53 and transmits it to the terminal unit 9. The publishable imagelist contains the file names and the file sizes of the whole image andreduced whole images may be pasted to it. The publishable image list isdisplayed on the terminal display 10 by way of the network 8 and theterminal unit 9.

Then, in Step S63, the user selects the whole image he or she wants outof the publishable image list. The terminal unit 9 transmits a wholeimage transmission request C1 to the monitor 5 in response to theoperation of selecting the whole image on the part of the user.

In Step S64, upon receiving the whole image transmission request C1, themonitor 5 reads out the whole image selected by the user and transmitsit to the terminal unit 9. The transmitted whole image is then displayedon the terminal display 10 by way of the network 8 and the terminal unit9.

Then, in Step S65, the user can easily identify the image regions in thewhole image being displayed on the terminal display 10 and drawing hisor her attention.

The monitoring system 1 accumulates the image signals of the wholeimages picked up in the past and thus can analyze the differences thatoccurred in the past as a post-processing operation. When analyzing thesituations where the differences occurred and the causes of thedifferences, the CPU 56 compares the luminance levels of different unitimages and draws user's attention to each unit image where a differencearises in luminance level to reduce the workload on the part of theuser.

1. A monitor for monitoring a panoramic image generated by shooting atarget area, sequentially shifting the shooting direction, the monitorcomprising: comparing means for sequentially detecting the difference ofluminance level between each unit image forming the generated panoramicimage and the corresponding unit image taken prior to the first unitimage along the same shooting direction and comparing the detecteddifference of luminance level with a predetermined threshold value;defining means for predefining one or more than one regions with athreshold value greater than the predetermined threshold value and/orone or more than one regions exempt from the comparison, on a unit imagebasis; and display control means for causing predetermined informationto be displayed on a display screen along with the generated panoramicimage according to the result of the comparison by the comparing means.2. The monitor according to claim 1, further comprising: shooting meansfor shooting the target area, sequentially shifting the shootingdirection thereof.
 3. The monitor according to claim 1, wherein thecomparing means compares the difference of luminance level on the basisof image compression unit.
 4. The monitor according to claim 1, whereinthe comparing means compares the difference of luminance level for eachprimary color signal or for each luminance signal and each colordifference signal.
 5. A monitoring method of monitoring a panoramicimage generated by shooting a target area, sequentially shifting theshooting direction, the method comprising: a comparing step ofsequentially detecting the difference of luminance level between eachunit image forming the generated panoramic image and the correspondingunit image taken prior to the first unit image along the same shootingdirection and comparing the detected difference of luminance level witha predetermined threshold value; and a display step of causingpredetermined information to be displayed on a display screen along withthe generated panoramic image according to the result of the comparisonin the comparing step; one or more than one regions with a thresholdvalue greater than the predetermined threshold value and/or one or morethan one regions exempt from the comparison being predefined on a unitimage basis in the comparing step.
 6. The method according to claim 5,wherein the difference of luminance level is compared on the basis ofimage compression unit in the comparing step.
 7. The method according toclaim 5, wherein the difference of luminance level is compared for eachprimary color signal or for each luminance signal and each colordifference signal in the comparing step.
 8. A computer program forcausing a computer to monitor a panoramic image generated by shooting atarget area, sequentially shifting the shooting direction, the programcomprising: a comparing step of sequentially detecting the difference ofluminance level between each unit image forming the generated panoramicimage and the corresponding unit image taken prior to the first unitimage along the same shooting direction and comparing the detecteddifference of luminance level with a predetermined threshold value; anda display step of causing predetermined information to be displayed on adisplay screen along with the generated panoramic image according to theresult of the comparison in the comparing step; one or more than oneregions with a threshold value greater than the predetermined thresholdvalue and/or one or more than one regions exempt from the comparisonbeing predefined on a unit image basis in the comparing step.
 9. Arecording medium storing a computer program for causing a computer tomonitor a panoramic image generated by shooting a target area,sequentially shifting the shooting direction, the program comprising: acomparing step of sequentially detecting the difference of luminancelevel between each unit image forming the generated panoramic image andthe corresponding unit image taken prior to the first unit image alongthe same shooting direction and comparing the detected difference ofluminance level with a predetermined threshold value; and a display stepof causing predetermined information to be displayed on a display screenalong with the generated panoramic image according to the result of thecomparison in the comparing step; one or more than one regions with athreshold value greater than the predetermined threshold value and/orone or more than one regions exempt from the comparison being predefinedon a unit image basis in the comparing step.